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(54) Colour correction with consideration of the scene illuminant 



(57) A color image editing system and method uses 
sensor responses in an image capture device to esti- 
mate and separate illuminant and surface reflectance in 
a scene. The system uses the estimated surface reflect- 
ances, along with a desired illuminant SPD, to calculate 
the corresponding XYZ values. To render the images on 
an imaging device , the XYZ values are converted into 
d ev j Ce -dependent display or printer values. The image 
data may be manipulated in a manner that is intuitive to 
humans, such as correcting the image data according 
to changes in lighting and surface color. 

An image capture device, such as a scanner or dig- 
ital camera, represents a scene as image data that is 
resolved as device-dependent channels, such as RGB. 
For each channel, the spectral reflectances of the sur- 
faces within the scene are described reflectance func- 
tions that are independnt of the lighting function that de- 
scribes the illuminant spectral power distribution (SPD). 
Display operators can independently adjust the surface 
and lighting functions. 
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Description 

The invention is directed toward image processing 
and more specifically towards correcting illumination in 
images captured by digital media, such as scanners or s 
digital cameras. 

Commercially available color image editing (CIE) 
systems are part of large graphics systems, such as 
desktop publishing systems that allow users to generate 
digital representations of color images, edit digital rep- 10 
resentations for visual appeal, and reproduce digital 
representations on different media. These systems in- 
clude image capture devices, such as digital cameras 
and scanners, image processing hardware, and color 
displays and printers; ' 15 

Today, most image capture devices have three color 
sensors which convert a multi-dimensional spectral sig- 
nal representing a scene into the output of three color 
channels: red (R), green (G), and blue (B). The RGB 
values of the image capture device are ultimately trans- 20- 
formed into device representations of the color display 
(RGB) or the color printer (CMYK). The methods for 
achieving this transformation are a critical part of any : 
color image editing system. ~ 

Prior art CIE systems are based on transformations % s 
of device-dependent RGB values into human-based 
representations, such as CIE XYZ. These intermediate 
XYZ values are then transformed into display RGB val- ' 
ues via a 3x3 transformation matrix or into printer CMYK 
values via a look-up table. This intermediate transf or- 30 
mation facilitates matches in color appearance between; 
the original image or scene and the rendered image -or 
scene. If the XYZ values in the original scene match the 
XYZ values in the rendered-scene, point by point* then 
the color appearance of the original and rendered scene 35 
will match. If the XYZ values'between the original and 
rendered scene do not match on a point by point basis, 
there is no guarantee that the two scenes will appear 
identical. - ' 

These prior art CI E systems have several disadvan- 40 
tages. First, XYZ rhatching.assumes that the illuminant 
in the original and the rendered scene are identical. 
While it may be possible to design image capture devic-; 
es that generate device RGB values that are within a 
linear transformation of XYZ values for surfaces under 45 
one illuminant, it is not possible to design' image capture 
devices that generate device RGB values that are within 
linear transformations of XYZ values for surfaces under 
any arbitrary or uncontrolled illuminant. Thus, the map- 
ping from the image capture device RGB; values into the so 
human-based XYZ values will always be accompanied 
by some information loss and, consequently, a color ap- 
pearance mismatch. .*-■*.■ 

Furthermore, prior art CIE systems do not separate 
the effects of the illuminant spectral power distribution 55 
(SPD) from the effects of the surface reflectances on the 
image capture sensors. As a result, they manipulate im- 
age properties derived from CIE XYZ, representations, 



such as hue, saturation, and brightness. These manip- 
ulations, however perceptual in nature, must be learned 
by operators. _ 

A color image editing system that allowed for color 
manipulation and adjustments based on illumination is 
desirable. It would be a further benefit if the manipula- 
tions and adjustments were intuitive to humans by cor- 
responding to change in lighting and surface color. 

Accordingly, the invention proposes a method for 
characterizing a scene as surfaces and illuminants com- 
prising the steps of : 

measuring an illuminant spectral signature of a cap- 
turing device; 

capturing the scene as an array of pixel values; 
characterizing a scene transfer function by combin- 
ing the illuminant spectral signature with a device 
signature of the capturing device: and 
characterizing a scene signature by describing illu- 
mination and surfaces corresponding to the array 
of pixel values according to the scene transfer func- 
tion. . ... 

A spectral-based color image editing (SBCIE) sys- r 
tern provides a general frame work for image correction, 
composition, and enhancement. An operator can cor- 
rect images that were captured under bad illumination, 
combine images captured under different illumination, 
or select illuminants that make the image visually ap-. 
pealing. Image manipulations are intuitive to humans 
because they correspond to changes tin light and sur- 
face color r -i .., . * , •:- ;v ■,. • r 

; A color image editing system.and method uses sen- 
sor responses of an image capture device to estimate 
and separate lighting and surface reflectances, in: a. 
scene. The system uses the estimated surface, reflect- 
ances, along with a desired illuminant spectral power 
distribution (SPD); to calculate, the corresponding XYZ 
values. To render the images on an imaging device, the 
XYZ values are converted into device-dependent, dis- 
play or printer values. . > : 

. : An image capture device, such as a scanner. or dig- 
ital camera, represents a scene as image data that is 
resolved as device-dependent channels, such as RGB. 
For each channel, the spectral reflectances of the sur- 
faces within the scene are described as functions that, 
are independent of a lighting function. The lighting func- 
tion describes illuminant spectral power, distribution 
(SPD). . ; 

First, an illuminant spectral signature of the image 
capture device is measured using a spectrophotometry 
sensor or the lighting function is estimated. Next, a 
scene is captured as image data. A scene transfer func- 
tion is generated as a combination of the lighting func- 
tion and the device signature .of the image capture de- 
vice. The image capture data is transformed according 
to the scene transfer function to generate a scene sig- 
nature that uniquely describes the scene without the ef- 
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feet of the image capture device. Display operators can 
independently adjust the surface and lighting compo- 
nents of the scene signature to achieve a desired image. 

The invention will now be described with reference 
to exemplary embodiments thereof shown in the accom- $ 
panying drawings, 1 in which: 

Figure 1 illustrates an embodiment for a SBCIE sys- 
tem. '- 

Figure 2 illustrates a flowchart for the SBCIE system 
shown in Figure 1 ; - 10 

Figure 3 illustrates a flowchart for step 50 shown in 
Figure 2. 

Figure 4 illustrates another embodiment for the SB- 
CIE system. 

Figure 5 illustrates a flowchart for the embodiment *5 
shown in Figure 4. 

Figure 6 illustrates an alternate embodiment for the 
SBCIE system. •■' 

Figures 7 A-B illustrate a flowchart for embodiment 
shown in Figure 6. 20 

A spectral-based color image editing (SBCIE) sys- 
tem provides a general framework for image correction, 
composition, and enhancement An operator can cor- 
rect images that were captured under bad illumination, 
combine images captured under different illumination, 25 
or select illumiriants ; that make the image visually ap- 
pealing. Image manipulations are intuitive to humans 
because they correspond to changes in light and sur-: 
face color. i " " »-••"■ 

the spectral power distribution- (SPD) of an illumi- 30 
nant can be described by *a single veclor, e, having.en^f 
tries which represent the amount of energy emitted oven * 
a range of wavelengths. For : each surface/ a spectral 
radiance factor, the light spectra that is emittedirom the, 
surface, has both diffuse ; and fluorescent components 35 
which are described as- a n by n matrix,' S, where n is 
the number of entries representing the function values - 
over a range'bf wavelengths. For example; n^= 31 when 
the function values are sampled between 400hm and 
700 nm in 10 nm steps. The diffuse components have 40. 
values solely along the diagonal vector s of the matrix- 
S.- Fluorescing components, which absorb light in one 
wavelength and emit light in* a longer wavelength, can- 
not be described by a single vector: 

' The combined spectral signature of multiple illumi- 45 
nants, e, can be approximated by a linear combination 
of a smaller set of spectral basis functions; Bj, where f: 
is the number of basis functions and w { are the weights * 
chosen to minimize the square error between the illumir : 
nant and its linear model approximation. s 0 

■'>■'':■. - •. f , . \ . , ■ 



The spectral signatures of diffuse .surfaces; s, may. 



be described by a linear combination of a smaller set of 
spectral basis functions. 

. f 
i = l 

These low-dimensional linear models of illuminant spec- 
tra and surface reflectances are efficient and reduce the 
amount of stored data required to reconstruct the spec- 
tral representations. 

Similarly, the spectral signature of a digital camera 
in terms of RGB values tends to be low-dimensional 
spectral data. The dimensionality of the image capture 
data limits the dimensionality of the spectral reflectance 
functions. The accuracy, of the spectral signatures of 
surfaces and illuminants is therefore limited by the in- 
herent dimensionality of the illuminant spectra and sur- 
face reflectances, the dimensionality of the image cap-, 
ture data, and current knowledge about surfaces and 
illuminant spectra. 

Figure 1 illustrates an embodiment for a SBCIE sys- 
tem 10. A spectrophotometry sensor 12 is connected 
to an image processor 14. The image processor 14 is 
further connected to a digital camera 16 and a display 
monitor 18. A printer 20 is connected to the image proc- 
essor 14 and the display monitor 18. . V 

In. this embodiment, the spectrophotometric sensor 
12; such as that disclosed by^ Vincent in U. S.P.N. 
5,144,498, is-based on a wedge. filter that records both 
the color signal and the illuminant SPD such that it is 
possible to estimate the surface reflectances.Tbe.digital 
camera 16 is a DCS200 : digital camera offered by the 
Kodak Company, The digital camera 16 has three sen- 
sors and a known device signature. r 

Figure 2 illustrates a process flow chart for the SB- 
CIE system 10 shown in Figure 1. The method for gen- 
erating representations of spectral reflectances in 
scenes is based on knowledge of or estimation of the 
illuminant spectral power distribution, knowledge of the 
spatial distribution and spectral sensitivities off camera 
sen sons,, f camera sensor responses for surfaces in the 
illuminated ,;scene, and assumption about the f : basis 
function for, surface spectral reflectances. t . 

■ In step ,11 Or the illuminants in a scene are stored as 
anTiHuminant spectral signature, e?. A preferred method 
for gathering. illuminant information ; is to directly meas- 
ure it using a spectrophotometric sensor. An alternate 
method is to restrict the dimensionality of theJinear vec- 
tor space for representing the illuminant SPD, .such as 
Buchsbaum disclosed in "A spatial processor model for 
object cojor perception", Journal of the Franklin Insti- 
tute, Vol 310, pp. 1-26, 1980, and then to estimate the 
illuminant SPD from the distribution of color pixels in the 
captured image. The performance of the estimation al- 
gorithms depend upon the number of classes of color 
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sensors. The larger the number of color sensor classes, 
the more accurate the illuminant SPD estimates are. 

In step 120, the illuminant spectral signature, e\ is 
transformed into a diagonal matrix, L, with nxn vectors. 
In step 130, the illuminant matrix, L, along with two 
stored functions that describe the spectral sensitivities 
of the digital cameras, T, and 3 spectral surface basis 
functions, B, are multiplied to create a scene transfer 
matrix, C, that describes all images captured by the dig- 
ital camera, T, where illuminant and surface information 
are decoupled. In this embodiment/ L is a nxn matrix 
having values solely along the diagonal, T is a 3xn ma- 
trix, and B is a nx3 matrix. The resulting scene transfer 
matrix, C, is a 3x3 matrix. In a general form that de- 
scribes f sensors, L is a nxn matrix, T is a 3xf matrix, B 
is a fx3 matrix, and C is a fxf matrix. 

.In step 140, a matrix A is created by inverting the 
matrix C. In step 1 50, an image of the scene is captured 
and stored as a 3xp matrix of sensor pixel values, R. In 
step 1 60, a nxp scene matrix, S; that describes the spec- 
tral reflectance of all points sampled, is constructed by 
multiplying 3 matrices: B, a nx3 matrix representing the 
3 spectral basis functions for surface reflectances, the 
inverted scene transfer matrix A, and the sensor pixel 
value matrix R. In this step, the system operator rhay 
change the spectral representations of surfaces in the 
scene, S, such as changes in the skin tone of a person, 
the color of a dress, the saturation of grass; or the color 
of drapes. ^ 

In step 170, the spectral representation of scene il- 
lumination may be altered manually or automatically by 
selecting a new scene illuminant vector, u\ The new 
scene illuminant vector, u', is transformed into ah- nxn 
diagonal matrix, L in step 1 80 ; L' is combined with S 
and H to calculate the XYZ values that should be dis- 
played or printed. When done manually, the illuminant 
SPD or surface reflectance can be changed until the im- 
age is visually appealing; such as in the color correction 
scheme proposed by Vrhel and Trussell in "Color cor- 
rection using principal "components", Color Research 
and Application, Vol. 17, pp. 329-337, 1992. In step 190, 
the image is rendered. ' ' 

Figure 3 illustrates a flowchart for rendering the 
scene. In step 191, a printer spectral signature and a 
display spectral signature are determined. In step 193, 
the scene function, S, is customized to account for the 
display spectral signature. In step 1 95 : the specific im- 
age may be rendered according to a look-up-table or 
specific function that corresponds to the printer spectral 
signature. 

Figure 4 illustrates an alternate embodiment for the 
SBCIE system 10'. The image processor 14 is connect- 
ed to the printer 20, the display^monitor 18, and a color 
scanner 22. Scanner RGB values are used to' estimate 
the weight factors of three reflectance basis f unctjons. 
The estimated weights and the corresponding spectral 
basis functions are then used to build three-dimensional 
linear representations of surface spectral reflectances. 



This embodiment may be generalized to scanners with 
f spectral channels, estimating weights for f surface re T 
flectance basis functions. 

Figure 5 outlines a process flow chart for the em- 

5 bodiment shown in Figure 4. 

In step 110, a color document is scanned, thereby 
creating a 3xp matrix of sensor responses, R. In step . 
120, a surface representation is constructed for each of 
the pixels in the scanned document by multiplying the 

10 3xp matrix R with a stored function, Y The matrix mul- 
tiplication in step. 120 can be stored in the scanner 
firmware as a nx3 matrix transformation applied to all 
scanner RGB values. 

■ One approach to matrix multiplication is as folbws. 

15 T is a 3xn matrix that describes the spectral responsiv- 
ities of a three-channel scanner for a range of wave- 
length samples. T corresponds to the point-by-point 
product of the spectral power distribution of the scanner 
illuminant, the ^spectral transmittances of apparatus 

20 within the optical path, and the spectral responsivity of 
the CCD scanner sensor. B is a nx3 matrix representing 
the 3 spectral basis functions for surface reflectances. 
C is a matrix created by multiplying T with B A is a 3x3 
matrix created by inverting C. Y is created by multiplying 

2S B and A. 

!n step 1 30, the spectral representation of an illumi- 
nant, u\ is selected and transformed into an nxn diago- 
nal matrix, L\ In step 140, L' is combined with S and H 
to calculate the XYZ values that should be displayed or 

30 -printed. • - : 1 

. Although this embodiment is based on a 3-channel 
output, as the number of spectral channels increases, ■■ 
the estimates of the spectral reflectances will improve. 
The estimates can be optimized by judicious selection 

35 of the spectral basis functions! - 

Figure 6 illustrates another alternate embodiment 
for the SBCIE system' 10". The image processor 14 is 
connected between the printer 20 and the display mon- ■ 
itor 18- Color images of scenes are often gendered on 

40 emissive or reflective displays. Even though image gen- 
eration is unknown; the operator has a. perception that: 
the displayed images are realistic depictions of; actual 
scenes'. One key area in computer graphics is devoted 
to developing realistic spectral representations of sur- 

45 faces and illuminants to be rendered on the display. One 
method is to have the operator select a region of the. 
scene that corresponds to a white surface Since a white 
surface has known spectral reflectance, an estimate of 
the illuminant spectral power distribution consistent with 

50 the displayed tristimulus values for the white surface can 
be made. " " *• ' ■ ''- ; * ' • ■ : 

--Figures 7A and B illustrate a flowchart for the em- 
bodiment shown in Figure 6. In step 110, a pixel region- 
that corresponds to a known surface, such as a white 

55 or 'neutral surface, is selected. The selection of this re- 
gion can be accomplished manually (by the user) or aur 
tdmatically (by a programmed rule, such as computing 
the mean RGB values over the entire region, as in the 
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"gray-world" assumption described by Buchsbaum). 
The mean RGB values of these pixels are stored as a 
3x1 vector w. In step 120, the display gamma lookup 
table (LUT) is used to convert w to linear RGB values, 
w\ In step 1 30, w* is mapped into the displayed tristim- s 
ulus values by multiplication with a stored 3x3. matrix 
generated by multiplying Z, a stored 3xn matrix contain- 
ing the CIE XYZ color matching functions, H, and a nx3 
matrix containing the spectral power distribution of the 
three display phosphors, M. . ■ - 1C 

In step 140, an illuminant, e\ is chosen that gener- 
ates the tristimulus values k when it is reflected from the 
known surface. One way is provide a database of illu- 
minant SPDs to find e' that minimizes the difference be- 
tween the predicted tristimulus values, k\ and the actual 
displayed tristimulus values, k. Instep 140, an illuminant 
e' is chosen from a database. In step 1 50, k' is computed - 
by multiplying e* with a nxn function representing the 
spectral reflectances of the known surface, S^, and the 
stored function, K In step 160, the predicted tristimulus 20 
values, k' and the actual displayed tristimulus values k 
are compared. If the difference between k and k' is less . 
than a stored threshold, e' is passed as a vector to step 
170 where it is transformed into a nxn diagonal ; matrix, 
L'. . 25 

In step 1 80, a scene transfer matrix, C, is generated 
by multiplying two stored functions: B, a nx3 matrix rep- 
resenting the surface spectral basis functions and H, a : 
3xn matrix representing the. CIE XYZ color matching / 
functions with L*. In step 90, the matrix C is inverted r t r 30^ 

In step 200,.the displayed 3xp RGB values for all p 
pixels; in the image are passed through the LUT,to gen- 
erate-RGB values-that are linear with displayed Jumi- 
nance. In step 210, the linear RGB values, R\ arecon n : : 
verted into XYZ values by multiplying R' with Z to gen- ; 35 
erate X. . . ... 

. In step 220;.a representation of spectral reflectanc- 
es of all points iri the image is. constructed. This is ac- - 
complished' by matrix multiplication of B, a stored nx3 
matrix, with A (the inverted sce;ne transfer matrix), and 40 
X (a 3xp matrix containing, the tristimulus values for all 
surfaces depicted in the scene). - . . 

In step 230, a new scene illuminant, u\ is chosen 
and transformed into a nxn diagonal matrix, L\ As in the. , 
previous.* embodiments,- the system, operator may *s 
change the spectral representations of surfaces in the ; 
scene, S, in step 220, or the spectral representation -of 
scene illumination, u". , • , ■ . 

I- In step 240, V is combined with S and H to calculate : . 
the XYZ values that should .be displayed or printed, so, 
These XYZ values are then sent to appropriate LUTs for 
subsequent display or printing. In step 250, the image 
is rendered: '. • . - : . * : ■ - i- 

Although the invention has been described in three 
channel RGB space, it would be obvious to -one skilled, , 55. 
in the art to increase the number of the channels r or ( .. 
translate the image data into a different visual represen- , 
tation such as CMY. - 



Claims 

1. A method for characterizing a scene as surfaces 
and illuminants comprising the steps of: 

measuring an illuminant spectral signature of a 
capturing device 110; 

capturing the scene as an array of pixel values 
150; 

characterizing a scene transfer function by 
combining the illuminant spectral signature with 
- a device signature of the capturing device 1 30; 
and 

characterizing a scene signature by describing 
-illumination and surfaces corresponding to the 
array of pixel values according to the scene 
transfer function 160. 

2. A. method for characterizing a scene as surfaces 
and illuminants as .defined in claim 1 further com- 
prising the step of rendering the scene 190. 

3. A method for characterizing a scene as surfaces 
and illuminants, as defined in claim 2, wherein the 
step of rendering further comprises the steps of:. 

customizing the scene signature to describe 
the surfaces in the scene 160; .. . : . . . , t 
customizing the scene signature according to 
.an effective illuminant level 170; and. . 
t . % ..rendering the scene signature 190^ , : ^ 

4. A method for, characterizing a scene as surfaces 
r and illuminants, as defined in claim 1 , wherein the 
. : step of capturing the scene comprises, the step of 
. scanning a rendered image po, .. , , 

5. . A method for characterizing, a scene as surfaces 

and illuminants, as defined in claim 3, wherein r the 
. step of customizing the scene signature to describe 
c the surfaces in the scene f urther comprises the step 

of characterizing the scene signature according to 

a look-up table 195. 

6. : A method for characterizing a scerie as surfaces 

and illuminants, as defined in claim 3, wherein the 
. step of customizing the scene signature to describe 
the surfaces in the scene further comprises the siep 
of rendering.the. scene surface matrix according to 
a transfer function that corresponds to a display de- 
vice signature. 193. , . . , , 

7. A method-for characterizing. a scene as surfaces 
and illuminants, as defined in claim .1, further com- 

- prising the step of capturing the device signature 

: j .130, . \ - 

8., : A method for characterizing a scene as surfaces 
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and illuminants, as defined in claim 1 , further com- 
prising the step of modeling a set of surfaces as a 
surface spectral signature 180. 

A method for characterizing a scene as surfaces $ 
and illuminants, as defined in claim 8, wherein the 
step of modeling a set of surfaces further comprises 
the steps of: 

scanning a rendered scene 110; *o 
generating an array of pixels that corresponds 
to the rendered scene 200; and 
determining a scene signature according to the 
array of pixels and the device signature 230. 

15 
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